Code sharing and reproducibility in survey-based social research: Evidence from a large-scale audit

Enlarge

Do researchers share their analysis code? And can published research findings be reproduced using this code?

In a new publication in Royal Society Open Science, Daniel Krähmer, Laura Schächtele, and Katrin Auspurg (LMU Munich) address these questions. Their study shows that results in the social sciences are often not reproducible—despite reproducibility being widely regarded as a fundamental prerequisite for the credibility and reliability of scientific research.

Based on more than 1,000 articles using data from the European Social Survey (ESS), the authors find that only 35% of researchers share their code upon request. Even when code is available, results are not necessarily reproducible. Of the 699 results examined in the study, only about half (51%) are numerically reproducible. The remainder is either technically irreproducible (23%) or yields different outcomes (26%).

Although these findings are sobering, there is reason for optimism.

Deviations between original and reproduced results are typically small and do not necessarily call the original findings into question. Moreover, many reproducibility problems could be avoided through better documentation and archiving of research materials—especially code and data. The study provides concrete recommendations for authors, editors, journals, and other scientific stakeholders to improve openness and reproducibility in social science research.

Krähmer, D., Schächtele, L., & Auspurg, K. (in press). Code sharing and reproducibility in survey-based social research: Evidence from a large-scale audit. Royal Society Open Science. https://doi.org/10.1098/rsos.251997

More from the LMU Newsroom